Learning to Translate: A Statistical and Computational Analysis

We present an extensive experimental study of a Statistical Machine Translation system, Moses [14], from the point of view of its learning capabilities, and we discuss learning-theoretic aspects of these systems, including model selection, representation error, estimation error and hypothesis space. Very accurate Learning Curves are obtained, by using high-performance computing, and extrapolations of the projected performance of the system under different conditions are provided. The experiments show that the representation power of the system is not currently a limitation to its performance, while the inference of its models from finite sets of i.i.d. data is directly responsible for current performance limitations. Of the models, the composition of the translation tables is more important that the numeric estimates of probabilities. The rate of improvement with sample size is no faster than logarithmic, and this is not likely to change with more advanced methods for estimating the numeric parameters. The fundamental limitation to the performance of the system seems to be a direct consequence of the Zipf law governing textual data. A few possible research directions are discussed as a result of this investigation, most notably the integration of linguistic rules into the model inference phase, and the development of active learning procedures.